Concerning the problem of unsatisfactory feature extraction and low recognition rate caused by redundant words in clustering dictionary in the practical monitoring video for two-person interaction recognition, a Bag Of Word (BOW) model based on improved Spatio-Temporal Interest Point (STIP) feature was proposed. First of all, foreground movement area of interaction was detected in the image sequences by the intractability method of information entropy, then the STIPs were extracted and described by 3-Dimensional Scale-Invariant Feature Transform (3D-SIFT) descriptor in detected area to improve the accuracy of the detection of interest points. Second, the BOW model was built by using the improved Fuzzy C-Means (FCM) clustering method to get the dictionary, and the representation of the training video was obtained based on dictionary projection. Finally, the nearest neighbor classification method was chosen for the two-person interaction recognition. Experimental results showed that compared with the recent STIPs feature algorithm, the improved method with intractability detection achieved 91.7% of recognition rate. The simulation results demonstrate that the intractability detection method combined with improved BOW model can greatly improve the accuracy of two-person interaction recognition, and it is suitable for dynamic background.